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ANALYSIS OF COVARIANCE FOR NONRANDOMIZED DATA 



Charles E. Hall 
Educational Testing Service 

Abstract 

The technique of analysis of covariance is examined for its relevance 
to educational and psychological investigations. A study of the arithmetic 
procedures, history, and statistical properties shows that, though the 
concepts are relevant to many data collection designs, the customary calcu- 
lations are often inappropriate. Some suggestions are made relevant to 
altering the arithmetic procedures or the data collection design under 
some common conditions encountered in educational and psychological studies. 



ANALYSIS OF COVARIANCE FOR NONRANDOMIZED DATA 



Charles E. Hall 
Educational Testing Service 

Considerable attention has been given lately to analysis of covariance, 
particularly in educational research where several factors of factorial 
design are fixed and some factors (or none) are randomized. The journal, 
Biometrics , dedicated the September 1957 issue to the problem of analysis 
of covariance. Recently, Lord (1967) presented the problems of this type 
of analysis as a paradox, and Campbell and Erlebacher (1970) presented some 
of the follies in use of the procedure. It is the intention of this paper 
to review the purpose, underlying assumptions, and calculation procedures 
and to arrive at a means of handling analysis of covariance when the 
classical assumptions are not met. 

Definition of the Problem 

As expressed by Cochran (1957, p- 262), a covariate "is a measurement 
taken on each experimental unit before treatments are applied, which is 
thought to predict to some degree the final response .. .on that unit." In 
experimentation, a covariate is usually a measurement taken of some factor 
which influences the experiment but cannot be controlled. 

The analysis of covariance is a statistical technique for eliminating 
the effect of the covariate on the response variate(s) being studied. 

The mathematics of the analysis of covariance is somewhat complex and requires 
using the figures shown, in Table 1. In the table, S denotes Sum, H, Hypothesis, 
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E, errors, T, totals, Y, the response variate, X, the covariate, and an * 
denotes an "adjusted" sum of squares. 3 denotes the various regression 
coefficients . 



Table 1 

Various Sums of Squares Used in Analysis of Covariance 



Source 


Variate 


Cross-product 


Co variate 


Regression 

Coefficient 


Hypothesis 


S H 


S H (xy) 


S H (x 2 ) 




Errors 


S E (y 2 ) 


S E <xy) 


S E (x 2 ) 


b e 


Total 


s T (y 2 ) 


S T (xy) 


S T (x 2 ) 


6 t 



As is customary S^ = S^, + S^ 

The calculations usually proceed as follows: 

First calculate the adjusted sum of squares for totals 

S* (y 2 ) = S T (y 2 ) - S 2 (xy)/S T (x 2 ) 

designating 

S T = S T (xy)/S T (x 2 ) . 

Then calculate the adjusted sum of squares for errors 
S E (y2) = S E (y2) " S E (xy)/S E (x2) 

^E = ^E ^ /^E ^ X ^ 



designating 
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Then find 

* p * p # p 

S R (y 2 ) = S T (y 2 ) - S E (y 2 ) . 

3 h = S H (xy)/S H (x 2 ) 



is not calculated ordinarily. 

The appropriate statistical test is then formed, 

F = const. S R (y 2 )/S E (y 2 ), 

the probability of F found and a decision made. 

This method is espoused by Cochran (1957), Scheffe (1959), Rao (1965) , 
and most others and will be called here the classical method. The method 
was apparently devised by Wishart and Sanders (1935)- It was first reported 
by Fisher (1932). 



Mathematics 

In order to examine the mathematics of covariance analysis , we shall 
use the method of fitting constants as devised by Fisher (1932) and consoli- 
dated by Yates (1933) and Wilks (1938). 

Let us use as an example a design with two groups and a covariate. 

We construct two pseudo variables (dummy parameters or design variables), 
one which is 1 for all observations and serves to extract the grand mean 
(constant term) from the observations, and one which is +1 for all observations 
in the first group and -1 for all observations in the second group and serves 
to produce the calculations for the t test. 1 The covariate will be x and 



^ilks would have used three pseudovariables, the constant term, and 
two variables for group membership. The two group membership variables were 
set up as 1 for the instant group and 0 for all other groups . 
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the variate y . Each observation vector then consists of 



(1, X, ±1, y) . 

The sum of the cross-products matrix will then be 



N 


Ex 


n i " n 2 


Ey 




Ex 2 


Ex 1 - Ex 2 


Exy 






n i + n 2 


£y l - Ey ; 


( syrnmet 


ric ) 




V 2 

Ey 



We will assume orthogonality of design to make the discussion easier; 
this requires n. - = 0. Constant fitting is usually done by Gaussian 

reduction (sweep out). Constants are usually fitted in the order in which 
they appear in the observation vector only to simplify notation. 

To fit the grand mean to the second row of A (sweep out jj), form 
the regression coefficient Ex/n and make the following calculations 

Ex 2 - (Ex/n) (Ex) ; (Ex^^ - Ex 2 ) - (Ex/n) (n 1 - n £ ) , and 

Exy - (Ex/n) Ey 

The reduction process of any entry a^^ of A for any constant k 
is a.^ - (a^) * ^ is °" lTvr: *- ous "that the regression coefficient 



p 

Notation conventions: Subscripts denote sample 1 or sample 2, and lack 

of subscripts denotes all observations. For example N = n^ + n^ and Exy^ 
denotes the sum of cross-products over group 1. 
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for variable K on variable i 
the sweep of U from all of A 



ls \i /a kk 

giving 



For 



B = 



N 

0 

0 

0 



Zx 2 - (Zx) 2 /N Zx 1 - Ex 2 



n l + n 2 



( symmetric) 



later use we will show 



0 

Zxy - ZxZy/N 

Zy l " Zy 2 
Zy 2 - (Zy) 2 /N 



After sweeping the second row and column of B we arrive at a matrix 
C which has relevant entries 

C 33 = (n x + n 2 ) - (E Xl - Ix 2 ) 2 /(Xx 2 - (lx) 2 /»> 

C . (l yi - Iy 2 ) - [(ZXj^ - Zx 2 )/(Xx 2 - (Xx) 2 /»)][lxy - ZxEy/H] 

0 = It. y 2 - (Xy) 2 /H) - (Xxy - ZxZy/N) 2 /(Ex 2 - (Xx) 2 /N) . 



With sufficient labor, it can he seen that 

ss ^ * C 4U 
SS H = C 3 l *^ C 33 

a*nd 3 ip = *^24^22 * 

After sweeping row 3 out of C, what remains in the (4,4) position is 

* , 2v 

S E ^ 

The constant does not appear in this calculation sequence hut 

Hi 

can t)e calculated as follows • 
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Sweep row 3 of B out of rows 2 and 4 giving & matrix D with elements 
d 22 = (Ex 2 - Ux) 2 /N) - (Z X;L - Zx 2 ) 2 /(n 1 + n 2 ) 
d 2l+ * (Exy - ExEy/N) - [(Ex ] _ - Ex 2 )/N][Ey 1 - Ey 2 ] • 

Without too much labor it can be shown that 
P E = d 2U^ d 22 ’ 

Statistical Estimatio n of Regression Coefficients. 

The usual design restriction placed on the covariate is that it be 
"unaffected by treatment." This statement has led to several practices. 

One is that of tahing covariate measurements before experimental treatments 
are applied to the groups. When this is done it is expected that the subjects 

will be .signed at random to treatment groups. This will leave E X;l 

„ _ Vv = £ and e-Q except for 

approximately equal to Ex 2 so i 2 

random fluctuations . When subjects are assigned at rand om to treatments 
is safe to assume that covariate values are distributed at rand om to 
treatments also. This leads Cochran (1957, p.264) to state that 
important to verify that treatments have had no effect on x [x.e. y 
covariate]. This is obviously true when the x's were measured before 
treatments were applied." A caution is taicen by the observation that 
Cochran expects subjects to b e assigned at_random to treatments. When 
subjects are not randomly assigned it is obvious that covariate val 
not randomly assigned either and that covariate values may be affected by 
factors which influence the assignment of subjects to treatments. 
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Another practice is that of choosing groups which are pairwise matched 

on the covariate and assigning the members of the pairs to the two groups at 

random . This will give Ex^ - Ex 2 = e = 0 exactly. 

The effect of having covariates "unaffected by treatment" can be seen 

quite readily when we examine the arithmetic of the classical solution. The 

3 ... 

adjusted estimates for treatment effect are the regression coefficients 
C^/C^ used for sweeping row 3 out of row 4 of C. Writing this using 
Ex^ - Ex^ = e gives us 



We can see that as e 0 the adjusted estimate of treatment effect approaches 
the unadjusted estimate of treatment effecu. That is, the effect of the covari- 
ate on the treatment means and treatment sum of squares becomes negligible as 
the difference between covariate means becomes small. 

The error sum of squares can be shown to be 




(Ey 1 - Ey 2 ) - e*(Exy - ExEy/N)/(Ex 2 - (Ex) 2 /N) 



(n 1 + n 2 ) - e 2 / ( Ex 2 - (Ex) 2 /N) 



r 2 _ sy 2 _ ( Exy - ExEy/N) 2 
l, y o.o 



N Ex 2 - (Ex) 2 /N 



[ ( Ey x - Ey 2 ) - e(Exy- ExEy/N)/(E X 2 - (Ex) 2 - (Ex) 2 /N)] 2 



( n l + n 2 ) - e 2 /(Ex 2 - (Ex) 2 /N) 



or, using e throughout 



3 

These are not the ones usually reported, but they are the ones which 
are used in the analysis. 
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(ly ) 2 _ 



N 



Zx l Z *l\ 

Ixy i - 4: 



+ I Zxy ? - n 



EXglyA e(Ey 1 - Hy 2 ) 



[(Sy-L - ^y 2 ) - eUxy - ZxZy/N)/(Zx 2 - (Zx) 2 /N)] 
N - s 2 /(Ex 2 - (Zx) 2 /N) 



As e -* 0 this expression approaches 



Ey 2 - (Ey) 2 /N - - Zy 2 ) 2 /W - (Exy^^ - Sx 1 2 y 1 / n 1 + lx y 2 ~ Y ‘ X 2 T ‘ y 2^ n 2^ 2 



which is the adjusted error term. Combining this information about the 
adjusted error term with the above observation that the treatment sum of 
squares is unaffected by a covariate with no difference between group 
means, we come up with the familiar observation, "Covariance adjustment 
reduces the sum of squares for treatment very little while reducing the 
sum of squares for error considerably" (see Snedecor, 1956, p. 399)- This 
effect is also reflected in the common statement: covariance analysis is 

used to reduce errors.^ 

The effect can also be seen by comparing with 8 m • 

hi x 

R = 2t * _ Exy - Ex2y/N 

x *22 lx 2 - ( £x) /N 



This is another misstatement about analysis of covariance. The 
"reduction of error variance" is no reduction at all; it is the elimination 
of a portion of variance due to the covariate which has nothing to do with 
treatment variation. The "reduction" of the so-called "error sum of squares," 
then, is only another step in obtaining the error sum of squares: there is no 

"error sum of squares" until the covariate reduction has taken place. 
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d i Exy - ExEy/N - (Ex, - Ex 2 )(Ey 1 - Ey 2 )/N 

3 = -=— • — 

E d 22 Ex 2 - ( Ex) 2 /N - (EX x - Ex 2 ) 2 /N 

Exy - ExEy/N - eCEy.^ - Ey 2 )/N 
Ex 2 - (Ex) 2 /N - e 2 /N 

As e 0 , 3,p converges to 3g • In short, since 3^ estimates the 

population regression coefficient, 3 T estimates it also, hut only if e 
is statistically null ! 

Difficulties with regression coefficients . Unfortunately, the above 

figures somewhat obscure what is really happening in the relationship between 

3,p and 3g • It is much easier to relate the two when 3„ is written in 

terms of deviations from sample means. Let SCP denote the sum of cross- 

w 

products within samples and SS^ be the pooled sum of squares within 
samples . 

Using SCP w to denote the sum of cross-products of variables within 
groups it can be shown that 



SCP^(x,y) 

= SS (x) 
w 

SCP w (x,y) + (Ex 1 - Ex 2 )(Ey 1 - Ey 2 )/N 
T ss w ( x ) + ( Sx ! - ^ x 2 ) 2 / N 

SCP w + e(Ey x - Ey 2 )/W 

SCP + e 2 /N 
w 




This shows more clearly that as (Ex^ - Ex 2 ) = e gets small and 
vanishes, 3^ approaches 3^, • It also shows quite clearly that, since the 
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p 



numerator contains e and the denominator contains e* - , 6,j, may he unpre- 

dictahly different than 6 when e is large. 

These formulations show that the estimation of regression coefficients 
is affected by the differences of covariate means among the groups. 

Difficulties with contrasts . An obscure and possibly crucial event 
happens to the entry in classical covariance analysis when the differ- 

ence between covariate means is not null. When row 2 of B is swept out of 
the matrix 



which shows that is altered by the covariance adjustment when the 

difference between covariate means is not zero. 

This alteration is only a matter of scale on the dummy variable when 
only two groups are involved- When more than two groups are involved, the 
matter becomes serious. 

For the three-group problem with deviation contrasts, group sizes 
n^ , n 2 and n^ , B is a submatrix. 



After adjustment for the covariate, this submatrix becomes very complex 
algebraically. The (l,l) entry is 



= n 1 + n 2 - (Zx 1 - Zx 2 ) 2 /(Zx 2 - (Zx) 2 /W) 



n + n 2 - e 2 / ( Ex 2 - (Ex) 2 /N) 
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The (l,2) entry is 

n 3 - (Ex 1 - Ex 1| )(Ex 2 - Ex^/N 

and the (2,2) entry is even worse. If the differences among the covariate 
means are substantial, there is a substantial alteration of the contrasts 
which one set out to test. The result of this alteration is to produce 
a new set of contrasts among means which are unknown to the statistician: 
i,e,,the mechanics of analysis produce a statistical test of unknown 
origin. When the differences among covariate means are large, the actual 
analysis may be very different from the one expected. 

It is conceivable that serious alteration of contrasts could take place-. 
That is, suppose the statistician originally specified the deviation contrasts 
among means as - Y , , - Y , , - Y . (each group mean contrasted 

with the grand mean) , it is not inconceivable that the differences between 
covariate means would alter these contrasts to something like 2 Y^ - Y . , 

Y 2 - Y, and Y 3 - 3/1+ Y, 

These arguments lead to the conclusion that classical covariance 
analysis is not always applicable to all problems which are commonly 
perceived as analysis of covariance. In fact, Cochran (1957 5 P* 261+) 
states: "The F - test of treatments against error for the x variate 

[i,e., covariate] is helpful when there is doubt whether treatments have 
had some effect on x, n One can only conclude that if the covariate means 
are dispersed significantly over treatments, then classical analysis of 
covariance is inappropriate. 

When subjects are not assigned at random to treatments . When subjects 
are not assigned to treatments at random, either over the entire list of 
subjects for the experiment or within blocks of subjects, the above arguments 
show the danger of using classical analysis of covariance to determine the 
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results of treatment. The author would prefer to describe this kind of 
investigation as a survey and not as an experiment. The term "experiment , " 
it seems, should he reserved for those investigations in which there is 
an opportunity for the experimenter to assign subjects to treatments 
randomly somewhere in the process of data collection. In educational 
investigations random assignment rarely occurs anywhere in the process. 
Teachers and students are almost never assigned to treatments at random and 
schools are always chosen from a list of those which will cooperate. 

When random assignment of subjects to treatments is not performed, there 
is considerable likelihood that systematic differences between treatment 
groups have occurred. Any data collection for covariates then should 
certainly include F tests of covariate over the collection design to insure 
that covariates have not been influenced by treatments or choice of treatment 
groups . Cochran (1957, p. 264) suggests this even for randomized designs; 
Snedecor (1956, p. 397) implies as much when he notes that "the means (on 
the covariate) . , .differ little more than would be expected. ..." 

What is one to do if it is apparent that the covariate means differ 
from treatment group to treatment group? The author has searched the 
literature of covariance analysis and no one seems to have come up with a 
reasonable answer. Zelen (1957, p. 310) reacts to the problem in this way. 

Still another use for covariance analysis as pointed out 
by Bartlett (1936) is to adjust treatment effects for systematic 
differences between experimental units in a non-random experiment. 

In this situation if the effects of treatment differences disappear 
after adjusting for initial differences, then one can conclude 
that the treatments do not differ. However, if after adjusting 
for initial differences among experimental units the treatment 
differences still remain, then the conclusion that the treat- 
ments actually differ is not necessarily a valid conclusion. 

Although an exact functional relationship between the dependent 
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and independent variates may not be known, an approximate 
relationship may be sufficient to adjust for the effects of non- 
randomness. Thus, if differences among treatments do not 
disappear after making such an adjustment, then we may conclude 
(i) treatments actually differ or (ii) treatments do not differ, 
but the functional relationship is not known to a sufficient 
approximation to adjust adequately for the non-random nature of 
the experiment . The conclusion that treatments actually differ 
will only be valid if the explicit form of the adjustment makes 
use of a known functional relationship between the dependent 
and the independent variates . 

The solution to the problem is stated in Zelen's last sentence. It 
can be restated as "if you can’t find a good regression coefficient you 
don’t have an analysis." Or: "if the analysis of covariance cannot provide 

a good regression coefficient, you can’t use it." Or: "Find a good 

regression coefficient before you analyze." 

When subjects are not assigned at random to treatments, the search for 
a regression coefficient can pose a serious problem. To generate an appro- 
priate regression coefficient one should fulfill the following requirements 

1. The regression coefficient should be independent of experimental 

> 

d at a . 

In classical analysis of covariance the independence is obtained 
mathematically by partitioning total variance . Tests of experimental hypotheses 
are generated from other independent partitions of the data. In some experi- 
mental situations it may be necessary to collect data exclusively for obtain- 
ing an appropriate regression coefficient. 

2. The regression coefficient should be appropriate to the range of 
data covered by the experiment. 
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A large percentage of regressions are curvilinear over the extremes 
of the range of the data and if the experiment has a selected range 
within the extremes , the correlation may he linear having one value within 
the range of the experiment and other values in other ranges . A good 
example would he the relationships between reading ability and age. Before 
the age of six it is nearly zero, after age 30 it may again he zero, hut 
between ages 6 and l8 the correlation may he linear and nearly .90. 

3. The regression coefficient should he obtained from the same measures 
as those used in the experiment. 

This requirement should he self-explanatory. 

These requirements make several points quite clear about using analysis 
of covariance when the covariate is a pre-experiment measure of the criterion 
variable . 

(a) Classical analysis of covariance is an analysis of gains scores 
if treatment groups are chosen at random or matched on the covariate. This 
is because the regression between variate and covariate can be estimated 
from the data at hand. 

(b) Analysis of gains scores is an analysis of covariance with an 
external regression coefficient. 

(c) Analysis of difference scores is an analysis of covariance (or gains) 
where the regression coefficient is assumed to be 1.0. 

Finding the appropriate regression coefficient . There are numerous settings 
where analysis of covariance is an inappropriate technique. The typical educa- 
tional survey or n experiment” done without random assignment is one of these. 

In these surveys there are techniques which can be used without completely 
destroying the value of analytic procedures and probability statements even 
though the techniques may not be rigorous. 
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In an "experiment" where the covariate is a pre-treatment measurement 
of the criterion variable and the treatment groups are at least representative 
of some whole population (which can he defined without referring to the 
criterion) it is possible to locate one very usable regression coefficient: 
the test-retest reliability coefficient from test norms P when the norms 
have been obtained from the same population as used for the "experiment" 
and the time lapse between test and retest is similar to that used in the 
"experiment. " 

When test norms do not exist or test-retest reliabilities have not 
been determined and if treatment groups represent some whole population, 
the error regression coefficient is a reasonable substitute. Although it 
has been known that error regression coefficients are inappropriate almost 
since analysis of covariance was devised, the difference in probability of 
F statements is rather small as Cochran (l957> P* 275) shows. 

There are circumstances where regression coefficients must be designed 
into the data collection. Consider a survey of a remedial program, say 
reading, where all pupils who have less than a particular reading score are 
subjected to a six-month remedial program. Reading scores are taken pre- 
and post-treatment and the question is asked "What happened?" Without a 
regression coefficient only descriptive statistics are available. The within- 
treatment group regression coefficient is inappropriate because it would wipe 
out gains if used. The nontreatment group regression coefficient is inappro- 
priate because the range of scores has been truncated by removing the treat- 
ment group. Any regression coefficient developed from the whole group is 

^The pretest posttest regression would be a reliability problem if no 
treatment had intervened. 
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in appropriate because part of the group has been treated. About the only thing 
to be done is to find a nearby school with the same kind of student body and 
run the same testing schedule for the sake of finding a regression coefficient. 

A similar problem exists in evaluating headstart-type programs. Here, 
there may be no way of obtaining an appropriate regression coefficient 
for several reasons. 

(1) The headstart program may exhaust the population (say ghetto 
dwellers) and no control group may exist. This makes it almost impossible 
to do any comparative study at all. 

( 2 ) The testing materials may not have been normed, thereby eliminating 
any comparison with a normative sample. 

( 3 ) Choosing a comparative group from another social stratum may be 
unsatisfactory because of systematic differences in pre-treatment scores and 
change in regression coefficient due to curvilinear ity of regression in the 
extreme range of the testing materials. 

( 4 ) Choosing a comparable sample in a nearby area and running the same 
testing schedule to obtain a regression coefficient may be prohibitively 
expensive. 

Conclusion 

The article by Lord (1967) has stirrcrd considerable investigation into 
this area of statistical analysis > so much so that the problem has become 
known as Lord’s Paradox. Much has been written to explain the apparent 
paradox in analysis. It is this writer ! s opinion that there is no paradox; 
there is only a misapplication of classical analysis of covariance procedure. 

The primary function of sampling design in analysis of variance is to 
obtain independent estimates of all the population parameters needed in a 
statistical problem from one complex data collection. The purpose of this 
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investigation is to show that the traditional collection procedure is 
inadequate for estimating regression coefficients for the analysis of 
covariance and change when subjects are not assigned to treatments at 
random. Some suggestions have been made about compensating for this 



inadequacy. 
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